Method for supporting cell image analysis

ABSTRACT

A fluorescent image, and either a bright-field image or a phase difference image of a plurality of cells are acquired by a microscope. A position of each cell is specified in the fluorescent image, and then a cell region of each cell is specified using the bright-field image or the phase difference image. A fluorescent part other than the cell region is thereby excluded. A class of each cell is determined, that is, the cells are classified. Thereafter, based on these results, a report is created.

This application is a continuing application, filed under 35 U.S.C.§111(a), of International Application PCT/JP02/11624, filed Nov. 7,2002.

BACKGROUND OF THE INVENTION

1) Field of the Invention

The present invention relates to a genome analysis on a cell imagephotographed by a microscope or the like for biotechnological researchand development, drug manufacturing, and the like. More specifically,the present invention relates to efficiency improvement in the analysisby automatically clipping images of each cell from the cell image,specifying types of each cell, and displaying the images and the typesof each cell as a list.

2) Description of the Related Art

Recently, following an advancement of genome science, a study ofidentifying a protein localization pattern and observing a morphogeneticchange for a cell into which a cDNA is injected, that is, a study ofidentifying functions of a DNA by performing a quantitative analysis ona morphogenetic change induced by injection of the cDNA into the cell isconducted. There is, for example, a demand for determining geneexpressions, determining a class of a protein localization pattern, andperforming a screening to determine how a cell changes by injecting agene into the cell so as to confirm pharmaceutical effects.

To meet such a demand, therefore, a change in a stained or fluorescentlycolored cell is observed by a microscope. A system that automaticallycaptures an image during the observation is conventionally put topractical use. These conventional techniques, by means of a plate onwhich many wells (holes) are arranged for cell cultivation, enable cellsto be cultured in large quantities and to be observed and screened bythe microscope.

A microscope-based image capturing apparatus HTS-50 manufactured byMatsushita Electric Industrial Co., Ltd is one example of a product. Theapparatus has a function of calculating quantitative and numeric valuessuch as a flatness of a cell, a length of a neurite, and a stainednucleus from cell images. However, the apparatus does not analyzeindividual cells but a texture of an entire image or a total extensionof elongate structures. Besides, the system is generally manufactured onthe premise that an experimenter visually checks all raw images. On theother hand, Beckman Coulter Inc. (United States) put into practical usean apparatus and an application for analyzing microscope images.Similarly to the HTS-50, the apparatus and application of BeckmanCoulter are provided on the premise that an experimenter visually checksthe images.

Further, Japanese PatentApplication Laid-open No. H5-249102 is oneexample of the conventional techniques.

These conventional techniques have, as described above, a disadvantageof a need for a person to visually check images one by one in ascreening. With such a method for visually checking an experimentalresult and extracting individual cell regions manually, it isdisadvantageously, extremely difficult to process images in largequantities generated in experiments conducted for multiple cDNAs.

Specifically, although it suffices to obtain one result from one well,several images to several tens of images are actually generated from onewell. Since about one hundred wells are used in one experiment, severalhundreds to several thousands of images are generated in one experiment.Besides, even if the images are displayed with reduced sizes, anecessary cell region occupies only a small area (for example, 1/400) ofthe image. The images cannot be, therefore, displayed as a list withoutprocessing them. Moreover, a success probability of injecting cDNAs intocells depends on a type of a target cell, a type of a cDNA, or aproperty of a culture medium, and sometimes the injection is successfulonly one cell in a hundred cells. Thus, since the cells in which thecDNA is injected are present only sporadically, it disadvantageouslytakes lots of time and labor to locate these cells and to manually nameeach protein localization pattern and each cellular morphogenesis.

Furthermore, to analyze images and display a result, a uniformprocessing is conventionally performed on entire images. Therefore, theresult is often displayed for each image. Since the success probabilityof the cDNA injection is low in cDNA experiments, it is necessary todiscriminate cells injected with cDNAs from those non-injected withcDNAs. Further, because of multiple types of fluorescent patterns and afeeble pattern in a background, such as a noise, it is disadvantageouslydifficult to discriminate each cell pattern from the background. In somecases, all cells are fluorescently colored. In other cases, a part ofcells are fluorescently colored in a grainy manner. If the adjacentcells are entirely fluorescently colored, they need to be separated fromeach other. If the cells are fluorescently colored in a grainy manner,it is necessary to recognize a dark part between grains so as not toregard it as the background. Thus, even if the experimenter visuallychecks the result, it is often difficult to make a clear determination.As a result, it is disadvantageously impossible to improve experimentaland analytical efficiencies.

SUMMARY OF THE INVENTION

It is an object of the present invention to at least solve the problemsin the conventional technology.

An image analysis supporting method according to an aspect of thepresent invention includes accepting an input of a fluorescent image ofa plurality of cells adhering to a well of a well plate, and an input ofany one of a bright-field image and a phase difference image or boththat are equal in field of view to the fluorescent image; detecting aposition of each cell in the fluorescent image; determining a class ofeach cell the position of which is detected; and outputting a list thatincludes information on the class and an image of each cell that isclipped from any one of the bright-field image and the phase differenceimage on the basis of the position.

A computer-readable recording medium according to another aspect of thepresent invention stores a computer program that causes a computer toexecute: accepting an input of a fluorescent image of a plurality ofcells adhering to a well of a well plate, and an input of any one of abright-field image and a phase difference image or both that are equalin field of view to the fluorescent image; detecting a position of eachcell in the fluorescent image; determining a class of each cell theposition of which is detected; and outputting a list that includesinformation on the class and an image of each cell that is clipped fromany one of the bright-field image and the phase difference image on thebasis of the position.

An image analysis supporting device according to still another aspect ofthe present invention includes a first input unit that accepts an inputof a fluorescent image of a plurality of cells adhering to a well of awell plate; a second input unit that accepts an input of any one of abright-field image and a phase difference image or both that are equalin field of view to the fluorescent image; a detecting unit that detectsa position of each cell in the fluorescent image; a class determiningunit that determines a class of each cell the position of which isdetected; and an output unit that outputs a list that includesinformation on the class and an image of each cell that is clipped fromany one of the bright-field image and the phase difference image on thebasis of the position.

The other objects, features, and advantages of the present invention arespecifically set forth in or will become apparent from the followingdetailed description of the invention when read in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory view of one example of a hardware configurationof an image analysis supporting device according to an embodiment of thepresent invention;

FIG. 2 is a block diagram of one example of a hardware configuration ofan image storage PC and an image processing PC;

FIG. 3 is an explanatory view of a functional configuration of the imageanalysis supporting device;

FIG. 4 is a flowchart of a processing performed by the image analysissupporting device;

FIG. 5 is one example of a fluorescent image;

FIG. 6 is one example of a bright-field image;

FIG. 7 is one example of a phase difference image;

FIG. 8 is an explanatory view of one example of a plate for cellcultivation;

FIG. 9 is an explanatory view of one example of a photographic regionwhen a plurality of images is obtained from one well;

FIG. 10 is a flowchart of an image acquisition processing;

FIG. 11 is an explanatory view of a content of a cell position detectionprocessing;

FIG. 12 is another explanatory view of the content of the cell positiondetection processing;

FIG. 13 is still another explanatory view of the content of the cellposition detection processing;

FIG. 14 is an explanatory view of one example of cell positionsdetected;

FIG. 15 is an explanatory view of a content of a class determinationprocessing;

FIG. 16 is an explanatory view of one example of a result of the classdetermination processing;

FIG. 17 is a flowchart of a recognition result determination processing;

FIG. 18 is one example of a report screen;

FIG. 19 is another example of the report screen; and

FIG. 20 is exemplary images of intracellular protein localizationclasses.

DETAILED DESCRIPTION

Exemplary embodiments of an image analysis supporting method, acomputer-readable recording medium, and an image analysis supportingdevice will be explained in detail with reference to the accompanyingdrawings.

FIG. 1 is an explanatory view of one example of the hardwareconfiguration of an image processing apparatus according to theembodiment of the present invention.

A microscope control module 101 controls an optical microscope 100 tophotograph a fluorescent image, a bright-field image, and a phasedifference image. The microscope control module 101 is set so that acell, which is successfully injected with a fluorescent marker includedin the cDNA, reacts to fluorescence. The optical microscope 100 is setso that the fluorescent image, the bright-field image, and the phasedifference image can be photographed in the same field of view. Themicroscope control module 101 includes xyz stages, a fluorescent(mercury light) shutter, a halogen light shutter, and the like, andfunctions of, for example, replacing a fluorescent filter andcontrolling a CCD camera to set an exposure time.

An image storage personal computer (PC) 102 and an image processing PC103 are personal computers as examples of an information processingapparatus. The PCs can be servers or workstations. The image processingPC 103 is particularly preferably a supercomputer or the like having ahigh-rate processing ability. FIG. 2 is a block diagram for one exampleof a hardware configuration of the image storage PC 102 and the imageprocessing PC 103. In FIG. 2, each of the image storage PC 102 and theimage processing PC 103 includes a CPU 201, a ROM 202, a RAM 203, a HDD204, a HD 205, a flexible disk drive (FDD) 206, a flexible disk (FD) 207as one example of a detachable recording medium, an I/F (interface) 208,a keyboard 210, a mouse 211, and a scanner 212. A bus 200 connects therespective constituent elements to one another. Further, a monitor(display) 104 and a printer 105 are connected to the bus 200.

The CPU 201 controls the entire image analysis supporting device. TheROM 202 stores programs such as a boot program. The RAM 203 is employedas a work area for the CPU 201. The HDD 204 controls data to be read andwritten from and to the HD 205 under control of the CPU 201. The HD 205stores data written under control of the HDD 204.

The FDD 206 controls data to be read and written from and to the FD 207under control of the CPU 201. The FD 207 stores written data undercontrol of the FDD 206, and causes the data recorded in the FD 207 to beread by the image processing apparatus. The detachable recording mediumcan be a CD-ROM (CD-R or CD-RW), a MO, a Digital Versatile Disk (DVD), amemory card, or the like in place of the FD 207. The monitor 104displays such data as a document, an image, and function information aswell as a cursor, an icon, and a tool box. The monitor 104 is, forexample, a CRT, a TFT liquid crystal display, or a plasma display.

The I/F 208 is connected to a network 209 such as a LAN or the Internetthrough a communication line, and is connected to the other server orinformation processing apparatus through the network 209. The I/F 208functions to interface the network with an interior of the image storagePC 102 or the image processing PC 103 and controls data to be input andoutput to and from the other server or an information terminalapparatus. The image storage PC 102 or the image processing PC 103 isconnected to the other server or the information terminal apparatusthrough this I/F 208, such as a modem.

The keyboard 210 includes keys for inputting characters, numbers,various commands, and the like, and inputs data. The keyboard 210 can bereplaced by a touch panel input pad, ten keys, or the like. The mouse211 moves the cursor, selects a range, moves a window, changes a size,or the like. The mouse 211 can be replaced by such as a track ball and ajoystick, as long as the element includes same functions as those of themouse 211 as a pointing device.

The scanner 212 optically reads an image such as a driver image andfetches image data into the information processing apparatus. Thescanner 212 also includes an OCR function that enables the scanner 212to read printed information to be used as data. The printer 105, such asa laser printer or an inkjet printer, prints image data and characterdata.

FIG. 3 is a block diagram of one example of the functional configurationof the image analysis supporting device according to the embodiment ofthe present invention. In FIG. 3, the image analysis supporting deviceincludes a fluorescent image information input unit 301, a fluorescentimage information storage unit 302, a bright-field image information andphase difference image information input unit 303, a bright-field imageinformation and phase difference image information storage unit 304, acell position detector 305, a class determining unit 306, a recognitionresult determining unit 307, a report creating unit 308, a reportinformation storage unit 309, a display controller 310, and an outputcontroller 311.

The image storage PC 102 is functionally composed by the fluorescentimage information input unit 301, the fluorescent image informationstorage unit 302, the bright-field image information and phasedifference image information input unit 303, the bright-field imageinformation and phase difference image information storage unit 304, andthe report information storage unit 309. The image processing PC 103 isfunctionally composed by the cell position detector 305, the classdetermining unit 306, the recognition result determining unit 307, thereport creating unit 308, the display controller 310, and the outputcontroller 311. However, one PC can realize the both functions of theimage storage PC 102 and the image processing PC 103.

The fluorescent image information input unit 301 accepts an input offluorescent image information that is a photographed image of aplurality of cells adhering to a plurality of wells, to be explainedlater. The fluorescent image information storage unit 302 stores theflorescent image information the input of which is accepted by thefluorescent image information input unit 301.

The bright-field image information and phase difference imageinformation input unit 303 accepts an input of either the bright-fieldimage information or the phase difference image information in the samefield of view as that of the fluorescent image input by the fluorescentimage information input unit 301. Alternatively, the bright-field imageinformation and phase difference image information input unit 303 canaccept the input of both of them. A selection of the bright-field imageinformation or the phase difference image information to be input can bemade according to a state of the cell to be photographed or the like.The bright-field image information and phase difference imageinformation storage unit 304 stores the bright-field image informationor the phase difference image information the input of which is acceptedby the bright-field image information and phase difference imageinformation input unit 303.

The cell position detector 305 detects a presence position of a cell inthe fluorescent image on the basis of the fluorescent image informationthe input of which is accepted by the fluorescent image informationinput unit 301 and stored in the fluorescent image information storageunit 302. Namely, the cell position detector 305 compares a candidateregion of a cell in the fluorescent image with either an average cellmodel or an average background model, and evaluates a similarity betweenthem, thereby detecting the position of the cell in the fluorescentimage. Specifically, the cell position detector 305 calculates aprojection amount, that is, an inner product between a main componentvector of at least either the average cell model or the averagebackground model, and a feature vector of the candidate region,calculates an evaluation value by normalizing the calculated projectionamount, and detects the position of the cell in the fluorescent image onthe basis of the calculated evaluation value. The average cell modelincludes a main component vector, a distribution value, and an averagevalue of each cell class in a sample image. Likewise, the averagebackground model includes a main component vector, a distribution value,and an average value of a background class in the sample image. Adetailed content of a cell position detection processing will beexplained later.

The class determining unit 306 determines a class of the cell at theposition detected by the cell position detector 305. The classdetermining unit 306 calculates a projection amount by obtaining aninner product between a main component vector of a model of each proteinlocalization class, and a feature vector of the cell at the positiondetected by the cell position detector 305, calculates an evaluationvalue by normalizing the calculated projection amount, and determinesthe class of the cell on the basis of the calculated evaluation value. Adetailed content of a class determination processing will be explainedlater.

The recognition result determining unit 307 determines a recognitionresult for each image or each well on the basis of the class determinedby the class determining unit 306. A detailed content of a recognitionresult determination processing will be explained later.

The report creating unit 308 creates report information shown in FIG. 18or 19, on the basis of information on the class determined by the classdetermining unit 306 and either the bright-field image information orthe phase difference image information, the input of each of which isaccepted by the bright-field image information and phase differenceimage information input unit 303 and which are stored in thebright-field image information and phase difference image informationstorage unit 304, of the cell extracted on the basis of the presenceposition detected by the cell position detector 305. The report creatingunit 308 can create the report information including information on therecognition result determined by the recognition result determining unit307. In addition, the report creating unit 308 can change therecognition result determined by the recognition result determining unit307 when a command is input by an operator.

The report information storage unit 309 stores the report informationcreated by the report creating unit 308. The display controller 310controls the monitor 104 to display the report information created bythe report creating unit 308 or stored in the report information storageunit 309. The output controller 311 controls the printer 105 to printthe report information created by the report creating unit 308 or storedin the report information storage unit 309, or transmits the reportinformation to the other information processing apparatus connectedthereto through the network 209 by the I/F 208.

Functions of the fluorescent image information input unit 301 and thefluorescent image information storage unit 302 are specifically realizedby one of or all of the keyboard 210, the mouse 211, the scanner 212,and the I/F 208. Functions of the cell position detector 305, the classdetermining unit 306, the recognition result determining unit 307, thereport creating unit 308, the display controller 310, and the outputcontroller 311 are specifically realized by making the CPU 201 execute aprogram stored in, for example, the ROM 202, the RAM 203, the HD 205, orthe FD 207. Functions of the fluorescent image information storage unit302, the bright-field image information and phase difference imageinformation storage unit 304, and the report information storage unit309 are specifically realized by, for example, the RAM 203, the HD 205and the HDD 204, or the FD 207 and the FDD 206.

FIG. 4 is a flowchart of processing procedure performed by the imageanalysis supporting device according to the embodiment of the presentinvention. In the flowchart shown in FIG. 4, the apparatus photographscells to acquire the fluorescent image and any one of the bright-fieldimage and the phase difference image or both through procedures shown inFIG. 10, to be explained later (step S401). FIG. 5 depicts one exampleof the fluorescent image. In the fluorescent image, a fluorescent markeris applied to each cDNA, and a gene expression cell can be specified bythe fluorescent marker emitting a light in a cell. Even if the cDNAs areinjected, they do not enter all cells. As shown in FIG. 5, only nucleiof cells, into each of which the cDNA is successfully injected, emits alight.

FIG. 6 depicts one example of the bright-field image. The bright-fieldimage is photographed using an ordinary lens, and enables recognizing ashape (profile) of a cell. FIG. 7 depicts one example of the phasedifference image. The phase difference image is photographed using arefractive index of a lens, and enables recognizing even a content ofthe cell. The bright-field image has an advantage in that focusing canbe easily performed, while the phase difference field has an advantagein that the profile of the cell can be displayed more clearly than thebright-field image although it takes longer time to perform focusing.Whether one of the bright-field image or the phase difference image isused or the both are used simultaneously can be appropriately selectedaccording to experimental and analysis conditions. For example, thefollowing method is considered. After performing focusing using abright-field image, a fluorescent image is photographed. The lens ischanged to another lens in an in-focus state, a phase difference imageis photographed, and only the fluorescent image and the phase differenceimage are stored in the image storage PC 102.

A position of a cell is detected from the fluorescent image acquired atstep S401 (recognition samples (unknown class) 1111 shown in FIG. 11)(step S402). Using the bright-field image or phase difference imageacquired at step S401, a cell region of the cell, the position of whichis detected at step S402 is obtained in a bright field of view (stepS403). A fluorescent part other than the cell region is thereby excluded(step S404). A class of each cell specified by the cell region leftafter excluding the other fluorescent part (step S404) is determined,thereby determining a class of each cell (step S405). A report iscreated on the basis of the result (step S406), thus finishing a seriesof processing procedures.

FIG. 8 is an explanatory view of one example of a 96-well plate for cellcultivation. In FIG. 8, the 96-well plate has wells arranged in alattice fashion. Each well is allocated one of symbols A to H for alongitudinal direction and one of numbers 01 to 12 for a lateraldirection. For example, an uppermost left well is allocated “A01”, and alowermost right well is allocated “H12”. Accordingly, one plate has8×12=96 wells of “A01 to A12”, “B01 to B12”, . . . , and “H01 to H12”.

FIG. 9 is an explanatory view that depicts one example of a photographicregion when a plurality of images is obtained from one well, which is anenlargement of one well (“G12 well”) on the well plate. In FIG. 9, forthe G12 well, fluorescent images, bright-field images, and phasedifference images are acquired at a plurality of positions (01 to 08),respectively. During the photographing, positions are concentricallychanged, that is, the images are photographed in the order of 01→02→ . .. →08. Alternatively, the images can be acquired with the positionschanged not concentrically but spirally.

FIG. 10 is a flowchart of an image acquisition processing using the96-well plate. In FIG. 10, a focusing target is moved to a first well(A01), and the first well is made into a photographable state (stepS1001). An M counter and an N counter are reset (step S1002). The firstimage (the fluorescent image) is acquired by photographing the cell's inthe first well (step S1003), and the N counter is incremented by one(step S1004).

It is determined whether the N counter reaches a preset value, forexample, “8” (step S1005). If the N counter does not reach “8” yet (“No”at step S1005), it is determined whether a gene expression cell ispresent in the image acquired at step S1003 (step S1006). Whether a geneexpression cell is present can be determined by depending on, forexample, whether a fluorescent part is present on the fluorescent image.If it is determined that the gene expression cell is present (“Yes” atstep S1006), then one of or both of the bright-field image and the phasedifference image in the same field of view are acquired, and the Mcounter is incremented by one (step S1007). If it is determined that nogene expression cell is present (“No” at step S1006), nothing isperformed and the processing proceeds to step S1009.

It is determined whether the M counter reaches a preset value, forexample, “3” (step S1008). If the M counter does not reach “3” yet (“No”at step S1008), images are acquired at a next position (step S1009).Thereafter, the processing returns to step S1004 and the respectiveprocessing procedures at the steps S1004 to S1009 are repeatedlyperformed. If it is determined that the N counter reaches “8” at stepS1005 (“Yes” at step S1005) or if it is determined that the M counterreaches “3” at step S1008 (“Yes” at step S1008), the processing proceedsto step S1010.

At step S1010, it is determined whether the present well is a last well.If the present well is not the last well (“No” at step S1010), thefocusing target is moved to a next well (step S1011), and the processingreturns to step S1002. For the next well, the processing procedures atthe respective steps S1002 to S1009 are repeatedly performed. If it isdetermined at step S1010 that the present well is the last well (“Yes”at step S1010), a series of processing procedures are finished.

When three sets of images including the gene expression cell areacquired for each well by these processing procedures, furtherphotographing is stopped. This makes it possible to suppress acquiringunnecessary images, and to save a processing time, a storage capacity,and the like. Even if an image of the gene expression cell cannot beacquired, up to eight images are acquired and the focusing target ismoved to the next well. It is noted that settings of the M counter andthe N counter can be appropriately changed.

FIG. 11 is an explanatory view of a content of a cell position detectionprocessing. In FIG. 11, learning samples 1101 in large quantitiesprepared in advance are input. The learning samples 1101 include both abackground learning image and a cell learning image. Each of the samplesis divided (step S1102), a feature is extracted from the divided samples(step S1103), and the extracted feature is normalized to generate afeature vector (step S1104). Further, a learning processing is performedusing the generated feature vector (step S1105), thereby creating anaverage cell model 1100.

Meanwhile, the cells in the fluorescent images of recognition samples1111 are class-unknown cells, in other words, classes of the cells arenot determined yet. The recognition samples 1111 are input in the sameprocedures as those for the learning samples 1101. Each of the samplesis divided (step S1112), a feature is extracted from the divided samples(step S1113), and the extracted feature is normalized, thereby creatinga feature vector (step S1114). Using the average cell model 1100, anidentification processing is performed (step S1115), thus extracting theposition of the cell in each of the fluorescent images.

FIG. 12 is an explanatory view of the content of the cell positiondetection processing, and specifically, FIG. 12 depicts an average cellmodel generation processing. In FIG. 12, each of the target learningsamples 1101 is divided into a preset number of segments (step S1102).If each of the learning samples 1101 is divided into, for example, 4*4pixel regions, the divided learning sample (original image) 1101 isprocessed, as a preprocessing, using a maximum filter 1202 for replacinga maximum value of the 4*4 pixels by one pixel, and a minimum filter1201 for replacing a minimum value of the 4*4 pixels by one pixel. Bydoing so, the learning sample 1101 is compressed to a maxim image and aminimum image at a size of 1/4*1/4. 96*96 pixel regions of the originalimage can be thereby represented by a 1152-dimensional feature vectorhaving n=24*24 (=1152) density values as a feature amount.

The feature vector of the learning sample j, which is normalized (thatis, an average value is 0 and a distribution value is 1 among therespective feature amounts of all the samples) for creating the averagecell model, is assumed as x_(j)=(x_(j,0), x_(j,1), x_(j,2), . . . ,x_(j,1151)). The feature vector is thus generated (step S1104).

Further, a covariance matrix R of all the learning samples is assumed tobe represented by:R=E _(j)(x _(j) ·x _(j) ^(T)).An eigenvalue problem of this matrix, as represented by:Rφ _(k)λ_(k)φ_(k),is solved, thereby calculating an eigenvector φk (where 1<k≦n) and aneigenvalue λk (where λk>λk+1) (step S1203). An eigenvector φ1 for amaximum eigenvalue λ1 is a first main component (main component vector).

If it is assumed that an average vector of a cell class (number ofsamples: m₁) is m^((c)) and that of a background class (number ofsamples: m₂) is m^((b)), then a projection of a learning sample x^((c))in the cell class on the first main component is represented by¹x^((c))={x^((c)T)φ₁}, and a projection of the average vector m^((c)) onthe first main component is represented by ¹m^((c))={m^((c)T)φ₁}.

A distribution value of projections of all the learning samples in thecell class is represented by

$\sigma^{{(c)}2} = {\frac{1}{m_{1}}{\sum\limits_{i = 1}^{m_{1}}\left( {{{}_{}^{}{}_{}^{(c)}} - {{}_{}^{}{}_{}^{(c)}}} \right)^{2}}}$(step S1204), and a distribution value of projections of all thelearning samples in the background class is equal to that in the cellclass (step S1205).

FIG. 13 is an explanatory view of the content of the cell positiondetection processing. Specifically, FIG. 13 depicts an average cellmodel generation processing. In FIG. 13, the target recognition samples1111 are divided into a preset number of segments similarly to thelearning samples 1101 (step S1112). Each of the divided recognitionsamples 1111 is processed, as a preprocessing, using a maximum filter1302 for replacing the 4*4 pixels by one pixel of a maximum value ofthem and a minimum filter 1301 for replacing the 4*4 pixels by one pixelof a minimum value of them. By doing so, the recognition samples 1111are compressed to a maxim image and a minimum image at a size of ¼*¼.96*96 pixel regions of the recognition sample 1111 can be represented bya 1152-dimensional feature vector having n=24*24 (=1152) density valuesas a feature amount.

The feature vector of the learning sample j, which is normalized (thatis, an average value is 0 and a distribution value is 1 among therespective feature amounts of all the samples) for creating the averagecell model, is assumed as x_(j)=(x_(j,0), x_(j,1), x_(j,2), . . . ,x_(j,1151)). The feature vector is thus generated (step S1114). Theprocedures executed so far are similar to those for the processing onthe learning samples 1101.

Regarding a point of a coordinate (i, j) of an image subjected to thepreprocessing, if a feature vector of a 24*24 pixel region around thispoint that serves as a central point is xi,j, a projection (amount) ofprojecting the feature vector xi,j on the first main component (maincomponent vector) is calculated (step S1303). The calculated projection(amount) is normalized and a similarity evaluation is performed. Namely,a value obtained by dividing a length between the projection (amount)and a projection 1m(c) of the average vector m(c) in the cell class by adistribution σ(c)2 of the cell class is set as an evaluation value ofthe cell class at the coordinate point (i, j) (step S1304).

If the following relationship is satisfied, it is determined that thepoint on the coordinate (i, j) belongs to the cell class.{(X _(i,j) ^(T)φ₁−¹ m ^((b))}²/σ^((b)2) >{X _(i,j) ^(T)φ₁−¹ m^((c))}²/σ^((c)2)

A minimum value in a local region is detected on the basis of a presetaverage cell diameter (step S1305). As a result, the cell position isdetected. The cell positions specified on the fluorescent image areshown in FIG. 14. In FIG. 14, each position with “+” is the cellposition.

FIG. 15 is an explanatory view of an example of a class determinationprocessing. In FIG. 15, reference sign 1500 denotes each class, that is,a protein localization cell model, which is composed by average cellmodel information on each class. Types of the protein localization cellmodel are, for example those shown in FIG. 20. A cell model generatingmethod is the same as the average cell model generating method.

As processing procedures, a feature vector of a class determinationtarget recognition sample is generated (step S1501). The feature vectoris generated using all of or a part of various feature amounts, forexample, those related to density statistics, those related to edgeelement features, those related to shape features, those related totexture features, and those related to run-length statistics.

Among the feature amounts, those related to density statisticsspecifically include a luminance average of all the images, a luminancedistribution of all the images, an overall distribution of averages innon-background rectangular blocks, an overall average of distributionsin the non-background rectangular blocks, a luminance average ofsecondary differential images, a luminance distribution of the secondarydifferential images, an overall average of averages in thenon-background rectangular blocks, and the like.

Those related to edge element features specifically include an averageof neighborhood density differences, a distribution of the neighborhooddensity differences, a luminance average of phase difference edgeimages, a lateral dark part run-length average of the phase differenceedge images, a lateral bright part run-length average of the phasedifference edge images, a longitudinal dark part run-length average ofthe phase difference edge images, a longitudinal bright part run-lengthaverage of the phase difference edge images, a covariance of a densitycooccurrence matrix of the phase difference edge images, a horizontalstandard deviation of the density cooccurrence matrix of the phasedifference edge images, a vertical standard deviation of the densitycooccurrence matrix of the phase difference edge images, a samplecorrelation coefficient of the density cooccurrence matrix of the phasedifference edge images, a ratio of the horizontal standard deviation tothe vertical standard deviation of the density cooccurrence matrix ofthe phase difference edge images, and the like.

Those related to shape features specifically include the number oflabels, a sum of areas of labeling items, a sum of circumferentiallengths of the labeling items, an average luminance ratio of all imagesto a non-background part, an average area of the labeling items, anaverage circumferential length of the labeling items, an averagecircularity of the labeling items, an average complexity of the labelingitems, and the like.

Those related to texture features include a horizontal average ofdensity cooccurrence matrixes, a vertical average of the densitycooccurrence matrixes, a contrast of the density cooccurrence matrixes,a covariance of the density cooccurrence matrixes, a horizontal standarddeviation of the density cooccurrence matrixes, a vertical standarddeviation of the density cooccurrence matrixes, a power of the densitycooccurrence matrixes, a sample correlation coefficient of the densitycooccurrence matrixes, a ratio of the horizontal average to the verticalaverage of the density cooccurrence matrixes, and the like.

Those related to run-length statistics include longitudinal and lateraldark part run-length averages, longitudinal and lateral bright partrun-length averages, a ratio of the dark part run-length average to thebright part run-length average, and the like.

An inner product between an eigenvector of the cell model 1500 in eachclass and the feature vector generated at step S1501 is obtained,thereby calculating a projection amount (step S1502). The calculatedprojection amount is normalized using distribution and average values,thereby performing a similarity evaluation (step S1503). Since theseprocessing procedures are the same as those at the steps S1303 and S1304shown in FIG. 13, they will not be explained herein in detail. Higher n(where n is a natural number) eigenvectors are used instead of the maincomponent vector so as to realize a better classification of thelearning patterns. As a result of the similarity evaluation, arecognition result is obtained. The recognition result is a resultregarding to which of the protein localization classes, the each classdetermination target recognition sample belongs. Once it is determinedwhich of the classes the each class determination target recognitionsample cell belongs, the cell classification is performed. FIG. 16depicts that classes, which are recognition results of the respectivecells the positions of which are specified on the fluorescent image, aredisplayed in abbreviations. In FIG. 16, “MEM” indicates that this cellis a membrane.

FIG. 17 is a flowchart of a recognition result determination processing.The recognition result determination processing is intended to determinerecognition results for respective images or respective tests from theindividual cell recognition results. In the flowchart shown in FIG. 17,the recognition results for the respective cells are collected for therespective images (step S1701). The collected recognition result foreach cell is determined in view of a certainty factor acquired from arecognition algorithm and significance (causal nexus) of each class(step S1702).

For example, it is assumed that four types of classes A, B, C, and D arepresent and that certainty factor thresholds of the respective classesare TA, TB, TC, and TD. It is also assumed that the certainty factors ofthe respective classes are obtained as xA, xB, xC, and xD. It is furtherassumed that a dependency relationship is held among the classes B, C,and D, i.e., the classes are changed with passage of time from B to Cand from C to D (B→C→D). Based on these assumptions, if each certaintyfactor is smaller than the corresponding certainty factor threshold, anunknown pattern is output as the recognition result. If the certaintyfactors of the classes A and B are equal to or greater than thecorresponding certainty factor thresholds, the class having the greatercertainty factor is output as the result. If the certainty factors ofthe classes B, C, and D are all equal to or greater than the certaintyfactor thresholds, the class B based on which the dependencyrelationship is held is output as the result.

If the above content of the processing is applied to an example ofprotein localization patterns, a localization to endoplasmic reticulumis changed to a localization to Golgi and to a localization to membranewith the passage of time. Therefore, the localization to endoplasmicreticulum, the localization to Golgi, and the localization to membranecorrespond to the classes B, C, and D, respectively. A localization tomitochondria, for example, corresponds to the class A.

In this way, the results for the respective images are collected for therespective tests (step S1703). The result for the respective tests isdecided by majority (step S1704), thus finishing a series of processingprocedures. As explained so far, using a foresight that single solutionis included per image or per test, the recognition results (genefunctions) for the respective images or respective tests can beestimated.

FIGS. 18 and 19 are explanatory views for examples of the report screen.FIG. 20 depicts an example of an intracellular protein localizationimage, and depicts a list of class names and class abbreviationscorresponding to the respective image examples. FIG. 18 depicts a reportscreen created based on the recognition results obtained by the classdetermination processing explained with reference to FIG. 15. “A01 01FA” in “Field-of-view image” box indicates a well “A01” and aphotographic position “01”, and also indicates that 15 gene expressioncells are present on this fluorescent image. Further, bright-fieldimages and phase difference images cut out of this field-of-view imageare attached and abbreviations of determined classes are displayed.

FIG. 19 depicts a report screen created based on the result obtained bythe recognition result determination processing explained with referenceto FIG. 17. Reference sign 1900 denotes a representative image for eachwell. A list of items, i.e., an item “fluorescent part ratio” 1901 thatindicates a ratio of a fluorescent part to all fluorescent imagesphotographed for one well, an item “number of cells” 1902 of the allfluorescent images photographed for one well, an item “exposure state”1903, an item “automatic class type” 1904 that indicates a classacquired by the recognition result determination processing, an item“similarity” 1905 with the model, and an item “manual class type” 1906are displayed. Each of the items 1901 to 1906 can be displayed bysorting.

By changing an item of a manual class type display box 1907, the “manualclass type” can be changed. By doing so, if a visual recognition resultdiffers from the automatically determined recognition result, a contentof one of the results determined to be correct can be easily selected.In addition, by depressing an image display button 1908, eachrepresentative image can be changed to a desired image. By depressing a“print report” button 1909, a content which is currently displayed isprinted. By depressing an “OK” button 1910, the content is confirmed asdisplayed and stored in the report information storage unit 309. It isalso possible to read the already stored report from the reportinformation storage unit 309, change a content of the report thus read,and store the report again in the report information storage unit 309.

As explained above, according to the embodiment of the presentinvention, each gene expression (cDNA injected) cell of interest can beautomatically detected; the fluorescent image, the phase differenceimage, and the bright-field image of only the detected cell part can beobtained; the protein localization, the morphogenetic change, and thelike can be recognized from these images and selected and arranged inthe report, which is automatically created, on the basis of informationsuch as the certainty factor of the recognition result. Since theluminance range of each of the images is adjusted so as to facilitatechecking the image on a screen, the experimental efficiency is greatlyimproved by displaying, printing, and saving the image on the screen.

Further, according to the embodiment of the present invention, afluorescent part in a region corresponding to each cell is specified onthe fluorescent image. To do so, a fluorescent pattern and a backgroundpattern are learned beforehand. An “average cell model” is created usingthe fact that a target is a cell, and a candidate region correspondingto one cell is specified by evaluation of normalized vectors. Further, aprofile of one cell can be extracted while referring to the bright-fieldimage or the phase difference image at the same position. Thus, not theentire images but the individual cell regions can be clipped andanalyzed, and a result of the analysis can be displayed.

Moreover, according to the embodiment of the present invention, thelearning and recognition processing for recognizing the fluorescentpattern and the morphogenetic pattern for each cell can be used. Inaddition, such a method as an eigenspace method, a subspace method, or aneutral network can be applied. By thus providing the computer with thelearning function, the determination result can be automatically output.

The present applicant has actually made trial reports. According to theconventional technique, image data of about 700 megabytes is generatedper plate, and a luminance range or the like needs to be adjusted forevery visual check. As a result, it takes several hours (three hours ormore) to look over the reports (with no time of adding determinationresults). As for the catalog made by way of trial, by contrast, acapacity of the catalog is about 6 megabytes per plate, so that it takesonly a few minutes to look over the catalog and that even determinationresults can be added.

The image analysis supporting method according to the present embodimentcan be a computer-readable program that is prepared beforehand, and theprogram is realized by executing it by a computer such as a personalcomputer or a workstation. The program is stored on a computer-readablerecording medium, such as an HD, an FD, a CD-ROM, an MO, or a DVD, andthe computer executes the program by reading it from the recordingmedium. The program can be a transmission medium, which can bedistributed via a network such as the Internet.

As explained so far, the image analysis supporting method, thecomputer-readable recording medium, and the image analysis supportingdevice according to the present invention can promptly and efficientlyextract the position of the cell in the image photographed by themicroscope and the type of the cell, and automatically display theextraction result for respective cell regions of each cell as a list.The image analysis supporting method, the computer-readable recordingmedium, and the image analysis supporting device according to thepresent invention are therefore suited to improve the experimental andanalytical efficiencies.

Although the invention has been described with respect to a specificembodiment for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art which fairly fall within the basic teaching hereinset forth.

What is claimed is:
 1. An image analysis supporting method comprising:accepting an input of a fluorescent image of a plurality of cellsadhering to a well of a well plate, and an input of any one of abright-field image and a phase difference image or both that are equalin field of view to the fluorescent image; detecting a position of eachcell in the fluorescent image; determining a class of each cell theposition of which is detected; and outputting a list that includesinformation on the class and an image of each cell that is clipped fromany one of the bright-field image and the phase difference image on thebasis of the position.
 2. The image analysis supporting method accordingto claim 1, further comprising determining a recognition result for eachfluorescent image or each well on the basis of the class of each cell,wherein the list also includes information on the recognition result. 3.The image analysis supporting method according to claim 2, wherein therecognition result is changeable by an operator when the recognitionresult is incorrect.
 4. The image analysis supporting method accordingto claim 1, wherein the position of each cell in the fluorescent imageis detected by calculating a similarity between each candidate region ofthe cell and any one of an average cell model and an average backgroundmodel or both.
 5. The image analysis supporting method according toclaim 4, wherein the position of each cell in the fluorescent image isdetected on the basis of a normalized inner product between a featurevector of the candidate region and a main component vector of any one ofthe average cell model and the average background model or both.
 6. Theimage analysis supporting method according to claim 4, wherein theaverage cell model includes a main component vector, a distributionvalue, and an average value of each cell class in a plurality of sampleimages.
 7. The image analysis supporting method according to claim 4,wherein the average background model includes a main component vector, adistribution value, and an average value of each background class in aplurality of sample images.
 8. The image analysis supporting methodaccording to claim 1, wherein the class of each cell in the fluorescentimage is determined on the basis of a normalized inner product between afeature vector of the cell and an eigenvector of each class.
 9. Acomputer-readable recording medium that stores a computer program thatcauses a computer to execute: accepting an input of a fluorescent imageof a plurality of cells adhering to a well of a well plate, and an inputof any one of a bright-field image and a phase difference image or boththat are equal in field of view to the fluorescent image; detecting aposition of each cell in the fluorescent image; determining a class ofeach cell the position of which is detected; and outputting a list thatincludes information on the class and an image of each cell that isclipped from any one of the bright-field image and the phase differenceimage on the basis of the position.
 10. The computer-readable recordingmedium according to claim 9, wherein the computer program further causesa computer to execute determining a recognition result for eachfluorescent image or each well on the basis of the class of each cell,wherein the list also includes information on the recognition result.11. An image analysis supporting device comprising: a first input unitthat accepts an input of a fluorescent image of a plurality of cellsadhering to a well of a well plate; a second input unit that accepts aninput of any one of a bright-field image and a phase difference image orboth that are equal in field of view to the fluorescent image; adetecting unit that detects a position of each cell in the fluorescentimage; a class determining unit that determines a class of each cell theposition of which is detected; and an output unit that outputs a listthat includes information on the class and an image of each cell that isclipped from any one of the bright-field image and the phase differenceimage on the basis of the position.
 12. The image analysis supportingdevice according to claim 11, further comprising a recognition resultdetermining unit that determines a recognition result for eachfluorescent image or each well on the basis of the class of each cell,wherein the list also includes information on the recognition result.